Papers with self-supervised models
CompoundPiece: Evaluating and Improving Decompounding Performance of Language Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | Currently, there is no dataset containing compound and non-compound words across languages . however, current LLMs perform poorly on words tokenized unfavorably by subword tokenization. |
| Approach: | They propose to use a Wiktionary dataset to evaluate large language models on decompounding . they find that current LLMs perform poorly on words tokenized unfavorably . |
| Outcome: | The proposed model outperforms the best unsupervised models by 13.9% accuracy on average. |
Data or Language Supervision: What Makes CLIP Better than DINO? (2025.findings-emnlp)
Copied to clipboard
| Challenge: | CLIP outperforms self-supervised models like DINO as vision encoders for vision-language models (VLMs) but it remains unclear whether this advantage stems from CLIP’s language supervision or its much larger training data. |
| Approach: | Embedding analysis shows CLIP captures high-level semantics while DINO is more responsive to low-level features like colors and styles. |
| Outcome: | Embedding analysis shows that CLIP captures high-level semantics, while DINO is more responsive to low-level features like colors and styles. |
Automatic Speech Recognition and Query By Example for Creole Languages Documentation (2022.findings-acl)
Copied to clipboard
| Challenge: | CREAM project aims to provide linguists with new methods for language documentation based on automatic speech recognition and keyword-spotting. |
| Approach: | They propose to use one hour of annotated data to design an automatic speech recognition system for two Creole languages. |
| Outcome: | The proposed model is based on an hour of annotated data and is usable by linguists. |
Rethinking Denoised Auto-Encoding in Language Pre-Training (2021.emnlp-main)
Copied to clipboard
| Challenge: | Pre-trained models such as BERT have achieved success in learning sequence representations, but they tend to learn representations that are covariant with the noise of pre-training. |
| Approach: | They propose to train self-trained models to learn noise invariant sequence representations . they encourage consistency between original sequence and corrupted version via unsupervised instance-wise training signals. |
| Outcome: | The proposed model improves on 11 natural language understanding and cross-modal tasks and achieves 0.6% gain on GLUE benchmarks and 0.8% increment on NLVR2 . |
Do self-supervised speech models develop human-like perception biases? (2022.acl-long)
Copied to clipboard
| Challenge: | Recent advances in speech recognition and representation learning show that self-supervised pretraining is an excellent way of improving performance while reducing the amount of labelled data needed for training. |
| Approach: | They compare the representational spaces of wav2vec, HuBERT and contrastive predictive coding (CPC) with the perceptual spaces of French-speaking and English-speaking human listeners. |
| Outcome: | The proposed models capture fine-grained perceptual phenomena while supervised models are better at capturing coarser, phone-level effects and effects of listeners’ native language on perception. |
Meta-Adapter for Self-Supervised Speech Models: A Solution to Low-Resource Speech Recognition Challenges (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing self-supervised learning models can learn latent representations from large amounts of unlabeled data, but they are expensive to fine-tune. |
| Approach: | They develop a meta-adapter to obtain meta-initialized parameters for self-supervised models . meta-Adapters show better generalization and extensibility than traditional pretraining methods . |
| Outcome: | Experiments on common voice and FLEURS datasets show Meta-Adapter performs better on low-resource languages . authors show it can be used on 12 low-source languages, but it requires huge computational resources . |
EZ-VC: Easy Zero-shot Any-to-Any Voice Conversion (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Current voice conversion methods struggle in zero-shot cross-lingual settings . authors develop a method that can be used in zero shot cross-linguistic settings despite advances in technology . |
| Approach: | They propose a voice-conversion model that combines discrete speech representations with a non-autoregressive speech decoder. |
| Outcome: | The proposed approach excels in zero-shot cross-lingual settings even for unseen languages and accents. |
Speech Analysis of Language Varieties in Italy (2024.lrec-main)
Copied to clipboard
| Challenge: | Recent advances in self-supervised learning provide new opportunities to analyze Italy’s linguistic varieties using speech data alone. |
| Approach: | They propose to automatically identify the geographic region of origin of speech samples drawn from Italy's diverse language varieties. |
| Outcome: | The proposed model can identify regions from speech recording and improve classification accuracy and yields embeddings that distinctly separate regional varieties. |